The Integration of Vision and Haptic Sensing: a Computational & Neural Perspective

When looking at an object while exploring and manipulating it with the hands, visual and haptic senses provide information about the properties of the object. How these two streams of sensory information are integrated by the brain to form a single percept is still not fully understood. Recent advances in computational neuroscience and brain imaging research have added new insights into the underlying mechanisms and identified possible brain regions involved in visuo-haptic integration. This review examines the following main findings of previous research: First, the notion that the nervous system combines visual and haptic inputs in a fashion that minimizes the variance of the final percept and performs operations commensurable to a maximum-likelihood integrator. Second, similar to vision, haptic information may be mediated by two separate neural pathways devoted to perception and action. This claim is based on a set of psychophysical studies investigating how humans judge the size and orientation of objects. Third, a cortical neural system described as the lateral occipital complex (LOC) has been identified as a possible locus of visuo-haptic integration. This claim rests on functional imaging studies revealing an activation of LOC to both visual and haptic stimulation. We conclude that much progress has been made to provide a computational framework that can formalize and explain the results of behavioral and psychophysical studies on visuo-haptic integration. Yet, there still exists a gap between the computationally driven studies and the results derived from brain imaging studies. One reason why the closing of this gap has proven to be difficult is that visuo-haptic integration processes seem to be highly influenced by the task and context.